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Abstract: Although the randomized controlled trial has been regarded as the sine qua non in recent 
years, we argue that understanding contexts for teacher practice is a critical factor in the 
implementation of intervention, but has often been overlooked in educational research design. In 
this paper, we argue that randomized trials may not be ideal for educational contexts that are multi¬ 
faceted, complex and often part of other community-based initiatives. To make this argument, we 
describe a study from “the context out,” a different lens than is typical in education research. In 
doing so, we consider how varied contexts create both constraints and affordances for teachers to 
engage in an intervention. Examining how teachers engage with new curricular practices, we focus 
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closely on five early childhood centers in high poverty neighborhoods, using mixed methods to 
understand both the influence of the intervention as well as the conditions that either supported or 
thwarted changes in practice. Our findings suggest that context plays a major role in educational 
practice, which has important implications for approaches to research in the future. Given this 
reality, we argue that an understanding of the context in which an intervention takes place must be 
considered in any calculation of “what works” in classrooms. 

Keywords: Context, intervention, professional development, randomized controlled trials, 
early childhood education 

Contextos de practica docente: (re) pensando el papel del contexto en intervenciones con 
docentes de educacion infantil y nuevos enfoques para la lectura compartida 
Resumen: Aunque los experimentos controlados aleatorios han sido considerados como condicion 
sine qua non en los ultimos anos, argumentamos que la comprension de los contextos de la practica 
docente es un factor critico en la implementacion de intervenciones, pero a menudo se han pasado 
por alto en los disenos de investigacion educativa. En este trabajo, argumentamos que los 
experimentos controlados aleatorios pueden no ser ideales para los contextos educativos que son de 
facetas multiples, complejas y con frecuencia parte de otras iniciativas basadas en la comunidad. Para 
sustentar este argumento, describimos un estudio "fuera de contexto," un objetivo no tipico en la 
investigacion en educacion. Tenemos en cuenta como contextos variados crean tanto restricciones 
como posibilidades para que docentes participen en una intervencion. Examinamos como los 
docentes se comprometen con nuevas practicas curriculares, focalizando particularmente en cinco 
centros para la primera infancia en barrios con altos niveles de pobreza, usando metodos mixtos 
para entender tanto la influencia de la intervencion, as! como las condiciones para favorecer o 
dificultar cambios en las practicas. Nuestros hallazgos sugieren que el contexto juega un papel 
importante en la practica educativa, lo que tiene importantes implicaciones para pensar 
investigaciones futuras. Dados nuestros resultados argumentamos que la comprension del contexto 
en el que se lleva a cabo una intervencion debe ser considerada en el calculo de "lo que funciona" en 
las aulas. 

Palabras clave: contexto; intervencion; desarrollo profesional; experimentos controlados aleatorios; 
educacion infantil 

Contextos da pratica docente: (re)pensando o papel do contexto em interven§oes com 
professores da educa§ao infantil e novas abordagens para a leitura compartilhada 
Resumo: Apesar de experimentos controlados randomizados foram considerados como 
condiyao sine qua non, nos ultimos anos, argumentamos que a compreensao dos contextos de 
pratica de ensino e um fator critico para a implementa^ao de interven^oes, mas tern sido muitas 
vezes negligenciado em projetos de investigate* educacional. Neste artigo, argumentamos que 
os experimentos controlados randomizados podem nao ser ideal para contextos educativos que 
sao multiplos, complexos e muitas vezes parte de outras iniciativas facetas de base comunitaria. 
Para sustentar esse argumento, descrevemos um estudo "fora de contexto", um objetivo que nao 
e tipica na pesquisa em educa^ao. Consideramos como variados contextos criar ambos os 
constrangimentos e oportunidades para os professores a participar de uma intervenyao. 
Examinamos como os professores se envolver com novas praticas curriculares, com particular 
incidencia em cinco centros para a primeira infancia nos bairros com altos nlveis de pobreza, 
uso de metodos mistos para entender tanto a influencia da intervenyao, bem como as condiyoes 
para favorecer ou dificultar mudan^as as praticas. Nossos resultados sugerem que o contexto 
desempenha um papel importante na pratica educativa, que tern implicates importantes para o 
futuro pensamento pesquisa. Dadas nossas descobertas sustentamos que a compreensao do 
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contexto em que e realizada uma intervenyao deve ser considerado no calculo do "o que 
funciona" em sala de aula. 

Palavras-chave: contexto; intervencao; desenvolvimento profissional; experimentos aleatorios 
controlados; educa^ao infantil 


Introduction 

Since No Child Left Behind, evidence-based practice has emerged as the sine qua non to 
improve how the educational community approaches research, and how this research is translated 
into meaningful intervention within classrooms. Within this model, randomized controlled trials 
(RCTs) of specific interventions, from literacy to mathematics to behavior modification, have 
become the means for determining “what works” in education. As any visit to the What Works 
Clearinghouse highlights, the list of “what works” has become long over the last decades and far 
more comprehensive. 

Nevertheless, it is questionable whether the implementation of “what works” has led to 
instructional improvements. In fact, one could argue that we are still no closer to closing the 
achievement gap than we were some 15 years ago (Reardon, Greenberg, Kalogrides, Shores, & 
Valentino, 2013). To some educational researchers, the answer to this conundmm is that 
interventions are not being implemented with fidelity. To others, however, the lack of substantial 
improvements is not entirely surprising. Practice-based researchers have argued that the dynamic, 
real-time, real-world life of unique classrooms present a challenge to randomized controlled trials 
and the implementation of any intervention with so-called fidelity (Erickson, 2014; Schorr, 1997). 
Rather, classrooms require teachers who are prepared to be nimble, to respond to contextual 
circumstances and immediate student needs, activities not typically associated with a controlled 
experimental design where such factors are viewed as extraneous, and subsequently partialed out. 

Although the randomized controlled trial is a powerful research design, especially for 
interventions that are conceptually neat, they may not be ideal for educational contexts that are 
multi-faceted, complex and are part of other community-based initiatives. Knowing that something 
can work (e.g. RCT), may be fundamentally different from knowing how to make it work in diverse 
contexts. Both understanding the variation in contexts and educational outcomes, and responding 
effectively to those elements, is at the heart of the goal of improvement science (Bryk, 2016). Given 
this reality, it may be necessary to broaden our lens to determine what constitutes credible evidence 
to assess and understand whether, how and why an intervention may or may not achieve its desired 
results. In short, understanding the context in which an intervention takes place must be considered 
in any equation of “what works” in classrooms. 

In this paper we describe an intervention study from a different lens than is typically 
reported. Specifically, we propose to look from the classroom out, rather than the intervention in, 
considering how context creates both constraints and affordances for teachers to engage in practice. 
We theorize context as the social, institutional, political and personal factors that influence teaching 
and learning. This includes the geographical setting (neighborhood, street, school, and classroom), 
the institutional norms and district policies which may frame the expectations for teaching and 
learning, as well as the dynamic interactions within classrooms which help us to understand “the 
connected whole” in which teachers engage in professional practice (Cole, 1996, p. 135). From this 
perspective, contexts for teacher practice are constituted by dynamic, process-oriented interactions and 
factors that create constraints and affordances for how teacher teach. Constraints and affordances 
are attributes, or elements, of each unique context that create and/or limit opportunities for action 
and activity (Hammond, 2009; Kennewell, 2001). 
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Within this framework, we approach an understanding of the work of teachers as they 
engage with new professional practices within their unique contexts. As Erickson (2014) encourages, 
this knowledge plays an important role in designing curriculum and professional development that 
recognizes the complexities of teaching that exist beyond single interactions between teacher and 
student. Rather than “intervention in” approach, where our interest would be in seeing whether or 
not teachers implement the interventions with fidelity, our “context out” approach places primary 
importance on understanding how specific contexts (e.g. physical, social, institutional, political) for 
teacher practice are comprised and influence the potential uses of evidence-based practices. It may 
also suggest additional factors to consider in designing and implementing interventions under 
varying contextual conditions. 

Background and Policy Context 

Our work began with a rather typical scenario in educational research. Responding to a request 
for proposals, the second author received a grant in 2013 targeted to improving early literacy 
development for preschoolers in the highest poverty neighborhoods in New York City (Neuman, 
2013). The project was designed to implement the Books Aloud! program, a well-researched early 
literacy initiative between libraries and early childhood centers (Neuman, 1999) by creating a more 
integral connection between the school library in five child care centers (described below in greater 
detail). 

We invited four child-care centers, and one Head Start to participate in the project (see 
Tables 1 and 2). Each of these centers satisfied the criteria stipulated by the grant: they resided in 
the most severely economically depressed area of New York City; they included a small, but 
generally under-utilized library; and they served children, ages 3 through 5 who were eligible for free 
and reduced lunch. Because we were providing resources including additions to the library, we were 
able to quickly recmit administrators to agree to our intervention and data collection efforts across 
the five centers. 
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Table 1 


Site and Neighborhood Demographics 


Site 

Mitchell 

Mapledale 

Grove Hill 

Shadyside 

Guardian 
Head Start 

NAEYC 

Accredited 

No 

Yes 

No 

Yes 

No 

Type of Program 

CBO 

Preschool 

CBO 

UPK 

CBO 

Preschool 

CBO 

UPK 

Head Start 
UPK 

Number of Preschool 
Classrooms 

2 

3 

2 

2 

4 

Total Preschool 
Enrollment 

21 

58 

30 

33 

71 

Oversight Agency 

EarlyLearn 

EarlyLearn/ 

State 

EarlyLearn 

EarlyLearn/ 

State 

EarlyLearn/ 
Head Start 

Funding/ 

Accountability 

Mechanisms 

City 

City/State 

City 

City/State 

City/State/ 

Federal 

% of Students 
Qualifying for 
Free/Reduced Lunch 

100% 

97% 

100% 

100% 

100% 

Number of Children 
in Participating 
Classroom 

14 

17 

15 

17 

16 

Mean Age (Yrs) 

3.54 

4.48 

3.64 

4.45 

4.51 

Ethnicity 
(N eighborhood) 

67% AA/ 

31% H 

15% AA/ 
68% H 

63% AA/ 
34% H 

42% AA/ 
53% H 

75% AA/ 

19% H 

Flousehold Poverty 
Rates 

(N eighborhood) 

45.2% 

37.2% 

55.3% 

45.2% 

37.2% 


Note: All names for sites and persons are pseudonyms. Neighborhood level data obtained from 2013 Census. 



Education Polig Analysis Archives Vol. 24 No. 89 


6 


Table 2 


Participating Teacher Demographics 



Mitchell 

Mapledale 

Grove Hill 

Shadyside 

Guardian 
Head Start 

Name 

Damien 

Maya 

Lisa 

Maria 

Ms. Fordham 

Gender 

M 

F 

F 

F 

F 

Age 

25-34 

25-34 

35-44 

25-34 

45-54 

Race/Ethnicity 

White 

Latina 

African 

American 

Latina 

African 

American 

Education Level 

BA 

BA 

MA 

BA 

MA 

Years Teaching 

<5 

6-10 

16-20 

<5 

16-20 

Degree Area 

Elementary 

Elementary 

Early 

Childhood 

Early 

Childhood 

Early 

Childhood 


Note: All names for sites and persons are pseudonyms. 


The Centers 

The participating centers came from different small-sized communities in the county and 
represented Community-based Organizations (CBO), organizations funded through the 
Administration of Children’s Services. Typical of early childhood programs, centers also received a 
patchwork of funding from other sources, including “universal prekindergarten” school-district 
funding and federal dollars (Early Head Start; Head Start). 

Starting in 2012, however, as part of a city-wide quality improvement system, then Mayor 
Bloomberg created EarlyLearnNYC, a $486-million dollar initiative designed to merge these funding 
streams and raise the quality of child care services. EarlyLearnNYC also attempted to improve the 
coverage of care across the city by shifting the supply of contracted slots to neighborhoods that 
needed them the most (Hurley, Kramer, Rosenbaum, & Miller, 2014). Nevertheless, in the course of 
trying to cut bureaucratic red tape and reduce barriers, these policy reforms created new ones. For 
example, in bringing about sweeping changes to the contracted childcare systems, many 
longstanding childcare programs lost contracts, while other received funding to service 
neighborhoods where they had little or no history (Hurley et al., 2014). Dozens of programs were 
eliminated, and thousands of publicly funded childcare seats went unused. 

Our childcare sites existed in the midst of this policy shift. Three of our centers, Mitchell, 
Grove Hill, and Shadyside, were nestled within large high-rise public housing complexes and went 
from small to smaller during the project, teetering on the edge of financial collapse and no longer 
able to even adequately fill two classrooms of preschoolers. Mapledale, on the other hand, aided by a 
savvy administrator and grant-writer, was able to survive through outside funds. Only Guardian 
Head Start, located between two public housing towers, remained unscathed through the transition, 
although it was re-competing for its contract under Head Start reauthorization. 
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In addition to these policy changes each site had multiple and varied accountability 
expectations. For example, two of the centers were accredited by the National Association for the 
Education of Young Children (NAEYC), and subject to their standards, teacher-child ratios, and 
developmentally appropriate practices. Head Start, on the other hand, has its own Head Start 
Outcomes Framework, recently revised (2012) that included descriptions of three new domains for 
learning and guidelines for bring additional attention to school readiness. The funds devoted to the 
Universal PreK program were accompanied by new guidelines for meeting the Common Core State 
Standards for 4-year olds. 

All of these changes occurred to a workforce that was minimally paid (average salary 
$35,000), and in many cases, overworked (8-hour days, with 2 weeks of vacation yearly). Several 
teachers had their masters’ degrees in early childhood (23%); 46% had their bachelor’s degrees, and 
31% had high school diplomas and above (see Table 2). According to the teachers, none had 
received professional development related to any of these changes in guidelines and standards for 
EarlyLearnNYC, NAEYC accreditation, Universal Pre-K, or Head Start re-competition. These were 
the settings for our intervention. 

The Critical Nature of Context 

Although many researchers have begun to turn their attention to the role of understanding 
local contexts, it rarely has made its way into our research designs. Nevertheless, there is growing 
evidence that the contexts in which teachers teach including the policy context, the provisions made 
for conveying changes in policy through professional development, and support from leadership has 
a great deal to do with their ability to create high quality environments for their students (Dennis & 
O’Connor, 2012). 

Too often, unfortunately, conceptions of context are limited to specific physical or geographic 
settings, such as a neighborhood or a specific school. However, the physical site and geographical 
location of a classroom are but two elements of the context in which teachers engage with their 
professional work. Instead of asking “what” is the context?, Erickson & Schultz (1977) argue it is 
better to ask, “when is a context?” This approach pushes towards a dynamic understanding of 
context that considers how the interaction between sites of action, people and processes create 
spaces that influence how teachers practice. From this perspective, context is constituted by the 
interactions within (and across) physical spaces, rather than by that space or setting alone (Rogoff, 
1990). 

As such, the contexts in which teachers practice must be taken into account when we 
consider new approaches to teaching. For early childhood settings, in particular, where policies such 
as financing, licensure, and eligibility of services are tightly bounded to the preschool experiences of 
young children, it seems especially imperative to understand how context interacts with the complex 
realities of changes in practice (Graue & Ryan, 2009). To better understand this “context out” 
approach, we examined our own intervention from within the classroom to ask: 

• How do contexts provide for and/or limit opportunities for teachers to engage with new 
approaches to pedagogy? 

• In what ways do changes in policy and institutional contexts affect teacher practices? 

• Given the influence of various context-related factors, what implications might this have 
for designing intervention research? 
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Research Design 

Educational programs are implemented in real world contexts, with characteristics that are 
both unique to a particular context, as well as those that are shared across contexts (Klingner & 
Boardman, 2011). To provide for the most robust analysis, therefore, we addressed these important 
questions using a mixed methods design. Given the uniqueness of each center, we theorized that a 
single-case design approach might provide us with formative information for examining the 
potential changes in classroom practice. Single-case research typically has an applied focus, often 
used in novel approaches in unique settings (Neuman & McCormick, 1995). In this case, we used an 
A-B-A-B time-series withdrawal design, where A is baseline (or business-as-usual), and B is the 
intervention (Kratochwill et al, 2010). Single-case design can bypass an error often found in group 
comparison studies because each case serves as its own control. The underlying principle is that if an 
intervention is effective, it should be possible to visually see a change in the dependent variable from 
the period prior to intervention (i.e., baseline) to the period during and after the intervention. In our 
case, we quantified the number of utterance level exchanges as our dependent variable between 
participating children and teachers in each class during shared book reading. Prior to starting the 
intervention and during the intervention itself, we continued to measure the number of exchanges at 
regular time intervals. Repeated measurement taken at baseline and intervention reduces several 
threats to internal validity, such as maturation and instrumentation, and statistical regression because 
patterns illustrative of these problems would have been recognized at baseline when stabilization 
was established. 

We began our study by walking the neighborhoods surrounding our centers to get a sense of 
the community and the local resources that might be available to children and their families. For 
example, we identified the proximity to the public library, and whether or not one could walk easily 
to access its literacy resources, the availability of cable or broadband access for the local community, 
of bookstores, newsstands, and so forth. Our goal was to gather various forms of qualitative 
information that would allow us to better understand how these factors might create affordances 
and constraints for literacy development. 

We then looked within each center site to better understand how the library within the 
center was used (or not); the teachers’ use of these and other literacy resources within their 
classroom environments, and their curriculum planning and development. We visited classrooms to 
examine resources and to observe classroom routines. Given that all the centers reported using the 
same curriculum (e.g. Creative Curriculum), we conducted informal classroom observations to get a 
sense of how, and in what ways the existing curriculum was used. 

Finally, to examine the social validity of the intervention (Wolf, 1978), we conducted 
interviews with teachers. Together, these mixed methods provided us with multiple approaches and 
lenses to address our research questions. 

The Intervention 

The Books Aloud intervention was an adapted version of a highly successful early literacy 
intervention to promote shared book reading in preschools (Neuman, 1999). Books Aloud was 
designed to better integrate library services by: i) creating a lending library of high quality books; (ii) 
providing training to teachers, school leaders, and paraprofessionals on research-based early literacy 
activities through 10 text-sets using narrative nonfiction and information text; (iii) engaging parents 
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in reading to their child through a build-a-home library program; and (iv) improving children’s oral 
language skills and academic vocabulary (e.g. critical to understanding concepts and content in 
subject areas) as measured by curriculum-based assessments and standardized assessments in 
receptive and expressive language. 

In particular, the text sets were targeted to topics identified in the school district’s effort to 
align all curricular materials with the Common Core in preschool. Such topics included the human 
body, healthy foods, and other topics related to living things. Each text set included five books, 
carefully organized to introduce children to multiple genre: rhyming books, narrative nonfiction, and 
information books. Accompanying each text set were lesson plans, designed to scaffold and support 
teachers as they engaged the children in conversation during each read aloud. Lessons included the 
‘gradual release of control’ starting with call and response interactions that then lead to more open- 
ended exchanges. Each lesson plan included educative materials for the teachers, such as a definition 
of keywords and concepts, language tips, and strategies for ‘what to look for’ in gauging the 
children’s responses (Neuman, Pinkham, & Kaefer, 2015). 

Teachers were encouraged to use the materials during whole group, shared-book reading during 
morning meeting time, two to three times weekly for approximately 15-minutes. The goal was to 
enhance the frequency and quality of teacher-child interactions in texts that represented multiple 
genre and increasing levels of textual complexity. 

Participants 

With the permission of the administrator at each site, our next step was to select a volunteer 
teacher to pilot our instructional materials. Our purpose was to observe the potential of our 
intervention for enhancing the number of quality language interactions in classrooms during shared 
book reading events. Furthermore, we wanted to engage each teacher as an active participant in the 
research, having him or her give us feedback on the materials’ usefulness for their children’s needs. 
These teachers received a 2-hour, one-on-one training session prior to enacting the program, and 
ongoing support throughout the project based on their requests for support. 

Procedures 

We devised an observational tool (known as PRINT!) to examine how our text set lessons 
and materials might change the quality of the interactions within classrooms. Based on our previous 
research, our presumption was that lesson plans with suggestions for engaging children in 
conversations around books would promote more talk than we had observed in classrooms. The 
measure was designed to capture the number of cognitively challenging, interaction-based utterances 
during the shared book reading time based on previous research (Bova, 2011; Pollard-Durodola et 
al., 2011). These types of utterances included information-seeking and open-ended questions, 
defining word meanings, connecting words and concepts and labeling. 

Nevertheless, in our initial piloting of the measure, we observed only a limited number of 
interactions that could be easily identified as cognitively challenging. Consequently, we decided to 
focus on the frequency of conversational turns (e.g. teacher query-child response, as well as the 
other way around), assuming that increases in the number of turns might provide an initial indicator 
of an enhanced language environment in the classroom. 

We then held a day-long training session with research assistants to develop consistency in 
our coding of interactions. Following training, research assistants were given five videos to code 
independently. Inter-rater reliability determined by Cronbach’s alpha was .90. 
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After training observers and reaching reliability, we began the first phase of the A-B-A-B 
design. In this design type, a non-treatment phase is initiated until the behavior in question 
demonstrates stability. During this phase we established baseline by recording the number of 
teacher-child interactions during the business-as-usual storybook reading using the classroom 
discourse as our single unit (e.g. instead of the individual). Once this baseline (A) was established, we 
began our first text set. During this time, observers visited each classroom twice a week and 
recorded the number of interactions for the text set book reading (B) over a 2-week period. 
Following this period, teachers then returned to their traditional storybook reading activity, using 
books of their own choosing (A). In this manner, we withdrew the intervention with the assumption 
that the interactional behaviors would return to baseline. Therefore, our approach was to vary the 
experimental condition every two weeks, with teachers engaged in storybook reading in a business- 
as usual manner (A) for 2-weeks, followed by our Books Aloud text set approach (B) over the next 2- 
weeks. Throughout the experimental period, research assistants conducted observations twice per 
week. We randomly videotaped sessions at each site, and had other qualified observers code them to 
determine reliability. Reliability was high, estimated over .95. 

At the mid-point of the experiment, we conducted a fidelity checklist. Using an adapted 
form from our previous project (Neuman, Pinkham, & Kaefer, 2015), we examined the fidelity to 
the lesson plan (e.g. setting a purpose for reading; introducing vocabulary; engaging children in 
open-ended responses; discussion) using a six-item, four-point Likert-type scale ranging from 1 
(low-fidelity) to 4 (high-fidelity). Teachers varied in their enactment of the intervention (as we will 
describe below), from a low of 33% adherence to the lesson to a high of 90%. 

Towards the end of the year, we engaged in hour-long conversational interviews with 
teachers to examine the social validity of our intervention (Wolf, 1978). Specifically, we asked them 
to discuss the intervention, to describe some of the challenges that might have influenced their levels 
of engagement in the project, and to inform us about the lands of policy shifts that might affect 
their day-to-day activities. We also conducted final interviews with each site director to better 
understand their insights on our project and their teachers’ engagement with our intervention. 

Analytic Strategy 

Our context-out strategy began with the classroom. Using each classroom as a single-case, 
we graphed the total number of conversational turns at baseline then during the intervention, 
followed by a return to baseline once it was absent. These graphs provided an important lens to 
understand which teachers appeared to take up and engage with the intervention, as well as which 
did not. We then used visual inspection to examine the effectiveness of our intervention. With an A- 
B-A-B design, we would expect increases in teacher-child interactions during the intervention phases 
of the experiment (B) compared to the business-as-usual instruction (A). Next, to better understand 
how and why certain teachers were able to engage with our intervention, we created qualitative case 
summaries (Stake, 1995) of each classroom as a nested site of teacher practice within a childcare 
center, a broader neighborhood community and a larger policy and accountability framework. In 
these case summaries, we aimed to capture the dynamic factors of each site that constituted contexts 
for teachers’ practice from the interactions between teachers, directors, sites, and policies. 

Using Dedoose as an analytical tool, we joined our interviews, photographs of the 
neighborhoods, centers and sites, and video observations to the case summaries to create sets of 
data for each case. Coding within and between these sets, we worked to understand and identify 
both case specific and cross-case themes that influenced how our teachers engaged with the shared 
book reading curriculum. From this process, we wrote a case study memo for each site that reflected 
the context dynamics at play within each classroom, as well as larger themes at play across these sites 
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(Saldana, 2009, 2012). These memos became sites of further meaning-making, and allowed us to 
begin to see connections across the sites in terms of constraints and affordances for teacher 
engagement, the role of policy shifts on daily practice, as well as how varying dynamics within the 
contexts of their practice influenced the ways in which teachers used the intervention. 

Results 


The Effectiveness of the Intervention 

As shown in Figure 1, two of the classrooms showed some evidence of an increase in 
conversational turns resulting from the intervention. The teachers at Mapledale, and to a lesser 
extent, Mitchell both engaged in more teacher-child interactions during shared book reading when 
using the text-sets than not. During the final text set, however, there seemed to be some 
intervention fatigue with conversational turns essentially flat compared to business-as-usual. 



MITCHELL - Shared Book Reading Verbalizations 



Baseline " Baseline Text Set #1 Withdrawal Text Set #2 Withdrawal Text Set #3 Withdrawal Text Set #4 

figure 1. PRINT! Data from Mapledale and Mitchell 


At our other three sites - Shadyside, Grove Hill and Guardian Head Start - the graphs suggest little 
change over the course of the intervention, as shown in Figure 2. With the exception of the second 
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GUARDIAN HEAD START - Shared Book Reading Verbalizations 




Baseline Baseline Text Set #1 Withdrawal Text Set #2 Withdrawal Text Set #3 Withdrawal Text Set #4 


SHADYSIDE - Shared Book Reading Verbalizations 
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GROVE HILL - Shared Book Reading Verbalizations 



Baseline Baseline Text Set #1 Withdrawal Text Set #2 Withdrawal Text Set #3 Withdrawal Text Set #4 

^igure 2. PRINT! Data from Guardian Head Start, Shadyside and Grove Hill 
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text set at Grove Hill, there were few increases in the number of interactions between the children 
and teachers during the intervention period. These results were consistent with our analysis of the 
fidelity to implementation checklist. For example, none of these teachers were inclined to use or 
refer to the lesson plans provided, or to ask open-end questions. In these cases, the intervention did 
not appear to influence how teachers interacted with children and vice versa. Neither the multiple 
genre nor the suggested prompts to engage children more fully in the texts seemed to affect how 
teachers and children interacted in these shared book readings. 

These results highlight the potential dissimilarities among participants within intervention 
research, but once again, do not provide insights into why such differences occur. Within our 
intervention study, we began to look more closely at the dynamic contexts for teachers’ practice 
through the analysis of our qualitative data. In other words, instead of merely noting a failure to 
implement with fidelity , we sought to understand the reasons for the lack of implementation. Using 
evidence from these analyses, we next examined the potential constraints and affordances that may 
have influenced teacher engagement in our intervention. 

Constraints and Affordances 

Our qualitative analysis provided an important lens for understanding the contextual factors 
that might have influenced teacher engagement. Each center was grappling with policy and 
institutional changes emerging from EarlyLearnNYC, although in different ways and with different 
degrees of success. As shown in Tables 3 and 4, all of our sites had different combinations of 
constraints and affordances that reflect different leadership styles, the influence of different policies, 
community dynamics and varied resources. 


Table 3 
Constraints 



Rising Accountability 
Requirements / Reformulated 
Reimbursements 

Shifting 

Policy 

Context 

Site Instability 
(Low Enrollment & 
Teacher/Staff 
Turnover) 

Shifting 

Demographics 

Mitchell 

X 

X 



Mapledale 

X 

X 



Grove Hill 

X 

X 

X 

X 

Shadyside 

X 

X 

X 

X 

Guardian 

X 

X 



Head Start 






Constraints. Our observations revealed a number of constraints in the enactment of the 
intervention. 

Shifting Polly Contexts. The paramount constraint was the shifting policy context, specifically, 
the city-wide EarlyLearnNYC. Along with new funding requirements, the policy also implemented 
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new accountability measures aimed at improving the quality of care and learning of children in these 
sites. Consequently, the directors and teachers were focused on understanding and implementing 
these new requirements. Similarly, the policy context for Head Start was shifting, framed by the new 
model instituted under the Head Start Act of 2007 that required Head Start programs to re-compete 
every five years for continued funding. In addition, all five sites were newly funded under the new 
Universal Pre-K (UPK) program, and those classrooms had additional requirements and 
supervision. Together, these policies (some of which were still in formulation) created confusion and 
uncertainty among directors and teachers regarding standards, requirements and programs. 

Rising Accountability Requirements and Reformulated Reimbursements. Relatedly, under the new 
EarlyLearnNYC policy, there were new requirements for paperwork, and missed deadlines with 
consequences that varied by site (see Table 3). Throughout our project, teachers and site leaders 
repeatedly noted how the increasing accountability requirements influenced their days, shifting their 
work from teaching to paperwork. According to Richard, the director at Mapledale, “Our teachers 
have so many responsibilities beyond teaching. Like the assessments, and showing how they are 
meeting standards for the Department of Education, or EarlyLearnNYC standards. It’s a lot of work 
to protect their time to actually let them teach.” 

Four of our centers had multiple layers of accountability. In addition to the NYC 
requirements, two of our sites were accredited by the National Association of Early Childhood 
(NAEYC) with its accompanying requirements and standards. At these sites, researchers observed 
children’s writing and/or drawing work posted on bulletin boards with distinct sets of standards 
labeled on the object, each attesting to the multiple standards teachers were working under. As one 
teacher told us, “It wasn’t easy to link your intervention to all the different standards I’m required to 
teach. Sometimes all I dream about are standards.” Or in the case of Ms. Fordham at Guardian 
Head Start “My lessons are due on Thursdays, and I have to tell Anita (site director) what I am 
teaching and why from the standards, three kinds of standards, and then be able to show her when 
she comes to watch me.” 

Under the new reformulation of reimbursements, monies now were to follow the child. Two 
of our centers (Shadyside and Grove Hill) were especially affected; when children left the program 
classrooms, so did their funds, with the result that there were fewer funds to support a full-time 
teacher. As of the end of 2014, EarlyLearnNYC led to the closure of nearly 50 centers due to 
inability to support enough classrooms to make the site financially viable (Hurley et al., 2014). 

Site Instability. Changes in enrollment had visible ripple effects on teacher assignment and the 
stability of instruction. Classrooms were closed; teaching assistants and teachers were required to 
move into different classrooms to deal with fluctuating populations of children. In two sites, 
participating teachers were new to teaching preschoolers, having been Infant/Toddler teachers the 
previous year. Staffing instability from declining enrollment also affected the families in the 
neighborhood. Directors had to turn away interested families because there was neither enough 
children to fill a classroom nor available space to add new children to existing classrooms. 

As student enrollment declined, staff were laid off and overall working conditions 
worsened. At Grove Hill, assistant teachers threatened to quit. At Shadyside, new assistants came 
and went, leaving our participating teacher, Maria, constantly negotiating for more support in her 
classroom. Planned projects and activities (even field trips) were cancelled or delayed due to a lack of 
support staff. The staff turnover left little time to prepare for our intervention. 

Shifting Community Demographics. At Grove Hill and Shadyside, shifting community 
demographics also affected enrollment. Gentrification across Brooklyn had greatly altered the fabric 
of the neighborhood, displacing working-class African American families who had lived there for 
generations. As Lisa, our teacher at Grove Hill noted, “These families were like a backbone here - 
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working-class families who had resources, who had been here for a long time. Now they are leaving 
and the families that remain have a lot more needs.” 

On the other hand, Shadyside’s enrollment was experiencing a shift from serving 
predominantly African American families in a historically African American neighborhood, to 
serving a growing population of Latino children and families. This naturally affected enrollment, but 
it also affected programming and staffing. Teachers needed to adapt to the language needs of 
emergent bilinguals who had previously represented a minor constituency in their program. For both 
sites, shifting demographics paired with the new policies under EarlyLearnNYC created serious 
constraints for enacting a new intervention. 

Table 4 


Affordances 



Protected Teacher 
Preparation Time 

Community/ 

Site Resources 

Supportive 

Leadership 

Early Childhood 
Teacher Identity 

Mitchell 

X 

X 

X 


Mapledale 

X 

X 

X 

X 

Grove Hill 





Shadyside 





Guardian 
Head Start 

X 



X 


Affordances. Nevertheless, at three of the sites, certain affordances served as protective 
measures against the uncertainties that these new policy initiatives created. These affordances seemed 
to influence the extent to which our participating teachers were able to engage with and take up the 
intervention. 

Protected Teacher Preparation Time. At both Mapledale and Mitchell, teacher planning and 
preparation time was heavily protected, giving both Maya and Damien at least one hour each day to 
prepare for their lessons. In contrast, teachers at Grove Hill and Shadyside did not have any 
preparation time. Lisa, at Grove Hill, told us, “If one child doesn’t sleep, which almost always 
happens, or there is a staff meeting, then there is no time, and there isn’t any time for me during the 
rest of the day, so sometimes I could be ready, but mostly I had to just do my best.” 

Community!Site Resources. Three of our sites, Mitchell, Mapledale and Guardian Head Start 
were each part of larger community-based initiatives. These organizations provided additional 
financial and material resources, such as safety-nets for funding gaps and teacher trainings across 
sites. In addition, two of our sites, Mitchell and Mapledale, were situated within large buildings that 
offered a multitude of community services such as a senior center, an afterschool program and a 
community health center and gym. Both sites benefited from the cross-connection of programs. 
Bulletin boards were filled with children’s artwork about their families and their communities, and 
were reflected in conversations between children and teachers and in the daily curriculum. 

Supportive Leadership. Some directors expressed strong support for the abilities of the 
participating teachers. Damien’s site leader at Mitchell, indicated delight that “Damien is getting this 
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confidence. We know he is a great teacher, and he is starting to know that too.” At Mitchell and 
Mapledale, this respect was reflected in our interviews with the teachers. Maya and Damien both felt 
that they were being “invested in” as teachers. Their directors often expressed confidence and 
support in the teachers’ abilities to make decisions, use good professional judgment, and provided an 
environment in which the teachers felt comfortable in taking risks. In these settings, the teachers 
seemed to willing and even eager to take on new challenges, as evidenced in their higher fidelity to 
the intervention lessons. 

Early Childhood Teacher Identity. Relatedly, this affordance reflects the professional identities of 
participating teachers in the project. During observations, both Maya and Ms. Fordham would tell us 
explicitly why they made certain pedagogical or curricular choices within their classrooms. Their 
confidence and strong identities were reflected in their work as they engaged with the text sets. They 
brought their own ideas and approaches into their teaching, and integrated the themes throughout 
their daily practice. These choices reflected the teachers’ rich professional knowledge of their 
students and what they felt their students needed, sometimes despite accountability constraints 
(Woodrow, 2007). 

These constraints and affordances represent some of the contextual factors that influenced 
the enactment of our intervention. They reflect the political, institutional, specific site and classroom 
realities that our teachers were busy negotiating throughout our project. For some of our teachers, 
like Maya, the affordances overrode the constraints. But for others, the constraints were too 
overwhelming and did not allow them to concentrate on teaching. For example, teaching at 
Guardian Flead Start had become a grinding daily act of compliance. Ms. Fordham’s professional 
knowledge and decision-making power had long since been taken over by detailed lesson plans and 
high levels of accountability. The site director at Guardian was so focused on accountability that Ms. 
Fordham was fearful to use our text sets and accompanying lessons as anything more than a script, 
the exact opposite of our goal. 

For Maria and Lisa, our teachers at Shadyside and Grove Hills, respectively, their role was 
more of a childcare provider than a teacher. At these two sites, the constraints of shifting 
community demographics, and the rising accountability and overall lack of resources under the new 
EarlyLearnNYC policies created powerful roadblocks to teaching. As a result, their focus was mostly 
on making it through the day, hoping to stop the inevitable closing of their sites. 

Conclusion 

Our central finding in this study is that contextual factors outside of schools affect what 
goes on inside them. Such outside factors including the shifting policies of funding and 
accountability for early childhood centers seemed to influence teachers’ involvement and 
engagement with our intervention. Analyzing the individual variation through a single-subject 
withdrawal design, it was evident that some contexts provided more support for engagement than 
others. Subsequent analyses of the constraints and affordances in these contexts through our 
qualitative analyses helped our researchers to better understand the complicated issues faced by 
these centers. 

According to Klingner and Boardman (2011), at least some of the challenges in educational 
research such as the persistent achievement gap between students of color and white students, can 
be explained by a research gap, a failure to conduct different types of research best suited for 
addressing complex issues in our schools. Randomized controlled trials have been highly useful for 
developing knowledge that something can work. However, how to make an intervention actually work 
reliably for different subgroups of students and teachers working under very different conditions 
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requires designs that are more sensitive to diverse contexts and populations (Cobb, McClain, 
Lamberg, & Dean, 2003). Traditionally, educational researchers have undervalued the importance of 
these contextual conditions, preferring to control for these extraneous variables rather than to 
understand them. 

In contrast, in this study we sought to better understand these factors. Examining the social 
validity of our program (Wolf, 1978), our efforts were to understand how the intervention was 
actually being experienced by the teachers it was supposed to help. The intervention was clearly 
aligned with the Common Core, a focus of preschool efforts in New York City; it emphasized 
asking higher-order questioning techniques, a skill that was often targeted in quality reviews in 
centers; and it provided a library of informational texts to both teachers and the families in these 
classrooms. 

Nevertheless, our interviews indicated that it did not address those issues that were more 
central to these teachers. The political and policy context had changed dramatically, with new and 
additional standards, requirements, accountability and funding priorities. Four of our five centers 
were in jeopardy of being closed due to new and sudden declines in enrollment; our fifth center, 
Guardian Head Start, was required to re-compete for funding for the first time in over 30 years. 
Needless to say, our intervention could not address the issues that mattered most and that created 
the greatest challenges to their daily practice. 

Often intervention study results like ours are simply dismissed out of hand as a failure of 
teachers to implement interventions with fidelity (Erickson, 2014). Certainly, the teachers in our 
study stmggled to follow our intervention in the ways we envisioned in design sessions at the 
university. But merely describing the failure to reach fidelity does not explain why this is the case. It 
is not that fidelity doesn’t matter; it is that why an intervention is not working, or doesn’t matter to a 
teacher, needs to be better understood. Had our research stopped at merely evaluating the 
intervention, rather than scrutinizing the myriad of factors influencing their work, we would not 
have understood they ways in which policy shifts and other contextual factors were influencing 
teacher engagement (Dennis & O’Connor, 2012). 

From this “context out” perspective, researchers can support teachers’ instructional work 
with a better knowledge of the constraints and affordances at play, building on the knowledge of 
unique communities and sites of learning. This mixed method approach to understanding the role of 
context allowed us to better understand the ways in which policy shifts, such as EarlyLearnNYC 
were leading to a host of constraints on our teachers. While EarlyLearnNYC was implemented to 
both save money and improve care, as with so many policies, it “creat[ed]... a lifeworld... [with] 
consequences for all those involved in early childhood” (Ryan & Graue, 2009, p. 191). These 
consequences included a destabilization of learning environments for young children while the 
policy was transforming the ways in which provisions for care were made in New York City’s low 
income communities (Hurley et al., 2014). 

For our continuing work on Books Aloud, this research has powerful implications. First, it 
suggests that the meaningfulness of an intervention, even one with the best intentions, must be 
based on the participants’ point of view. We cannot merely assume that a new program regarded as 
useful to a school administrator or Center Director will have meaning to those who are charged with 
enacting it. Rather, we must begin to engage practitioners more fully with researchers and others in 
development, testing, and enhancing the clinical work of schooling. Second, it suggests that a better 
understanding of the context would have more likely led to an intervention that was better designed 
to address difficult problems that the schools and centers were facing. Too often, solutions come 
from outside in, rather than inside out. And third, schools and centers are complex organizations, 
with many different actors assuming many different roles. Understanding this context would have 
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helped us to take advantage of the enormous talents of individuals within the organization in ways 
that could have more effectively contributed to children’s learning and development. 

As Bryk (2015) has insightfully noted, accelerating how we learn to improve in schools will 
require us to view improvement as a science, and as a science, consider more thoughtfully how these 
contextual factors may contribute to outcomes. From a policy standpoint, this means a greater focus 
on developing systematic and organized methods with the explicit goals of improving outcomes. 

Too often, we have overlooked the necessary steps of learning from context and from the 
participants themselves. We certainly need to know what mechanisms work to help children and 
teachers create meaningful knowledge and learning experiences. However, as Jones, Pickard and 
Stronach (2008) note, educational research should be linked to “deep issues concerning learning and 
motivation, rather than tied to the evaluation of ephemeral initiatives in a naive kind of what works’ 
rationale in order to improve instructional outcomes in meaningful ways” (p. 27). These deep issues 
are often hidden from view when we do not know the contexts for teacher practice. Valuing what 
both “intervention in” and “classroom out” studies tell us about when, why, and how teachers take 
up evidence-based practices may inform “what works” approaches to improving educational 
outcomes. 

While randomized controlled trials may be considered the gold standard for policymakers, 
mixed-method approaches to understanding how and why interventions may or may not work 
within real classrooms may be a better way to make sure that meaningful instructional improvements 
are actually taking place (Cobb et al., 2003). Although the drive for “what works” is laudable, we 
must not marginalize the existing wisdom, operational know-how, and local knowledge that may 
allow us to create and implement effective, local, and sustainable interventions within the unique 
contexts of teacher practice. 
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